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' Abstract 

r ) , In this paper, we give a polynomial {0{n^)) algorithm for finding a 

. ■ longest common pattern between two permutations of size n given that 

i-J^ ' one is separable. We also give an algorithm for general permutations 

, whose complexity depends on the length of the longest simple permutation 

involved in one of our permutations. 

1 Introduction and basic concepts 

> . 

0> . The study of patterns in permutations has blossomed these last years: from a 

' combinatorial point of view with the recent proof of the Stanley- Wilf conjecture 

by Marcus and Tardos, and from an algorithmic one with the development of 
algorithms for pattern and general pattern involvement. Although the general 
\^ . pattern involvement problem is A^P— hard, some polynomial solutions exist for 

' special kinds of patterns like the separable ones [BBL98, Iba97]. In this article 

fH , we study the problem of finding a longest common pattern between two permu- 

tations (7i and (72 i-e. a permutation a which is involved in both permutations 

a"* . CTi and (72 . This is a generalization of the pattern involvement problem since 

finding if the longest pattern between cti and <T2 is equal to cti is equivalent to 
^ , the pattern involvement problem. 

First, we give a polynomial algorithm based on the work of [BBL98] for 
finding the longest common pattern if one permutation is separable. Then 
' we generalize this algorithm for general permutations. The complexity of our 

algorithm is highly based on the length of the longest simple permutation [Bri] 
involved in our permutations. 
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1.1 Permutations 

A permutation a of an interval / of N is a bijective map from / to itself. We 
denote by ct^ the image of i by a. The permutation a could either be seen as a 
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function or a word (TiCTj+i . . .Uj, where I = {h : i < h < . For example the 
permutation a = 142563 is the bijective function such that a{l) = 1, a{2) = 4, 

ij(3) = 2, 0-(4) = 5 In the following, when we consider permutations without 

giving explicitly the interval /, we mean that 7 = {1, . . . , n} for some n. 

Deflnition 1.1. A permutation tt = tti . . . tt^ is called a pattern of the permu- 
tation a = (7,+i . . . Ui+n oiI={h:i+l<h<i + n}, with fc < n, if and only 
if there exist integers i-\-l<i\<i2<---<ik<i + n such that cTj^ < o-j^ 
whenever tt^ < 7r„j. Wo will also say that tt is involved in a or that a contains 
77. The subsequence ai-^ai^ . . . (t,^ is called an occurrence of tt in a. 

A permutation a that does not contain tt as a pattern is said to avoid tt. 

Example 1.2. For example (t = 142563 contains the pattern 1 3 4 2 and 
156 3, 146 3, 2563 and 1 4 5 3 are the occurrences of this pattern in a. 
But a avoids the pattern 3 2 1 as no subsequence of length 3 of cr is isomorphic 
to 3 2 1 i.e. is decreasing. 

A number of enumerativc results has been proved on classes of pattern avoid- 
ing permutations for patterns of length 3, 4 and multiple patterns. More recently 
results about the algebricity of the generating function of general classes of per- 
mutations have been given [BHV06] . 

Another field of study of these permutations is from the point of view of 
pattern involvement. The problem of deciding if a permutation tt is a pattern 
of a permutation a is iVP— hard but this problem is proved to be polynomial if 
the pattern is separable [BBL98, Iba97]. 

Definition 1.3. A permutation a of size n is called separable if it avoids the 
patterns 3 14 2 and 2 4 1 3 or equivalently if it has a binary separating tree. 

Definition 1.4. A binary separating tree is a binary ordered tree with n leaves 
such that each internal vertex is labeled by -|- or — . 

For each such tree, there is a unique way [BBL98] to decorate its leaves 
(considering them from left to right) by cri, CT2j • ■ • , o"„ such that: 

1. (Ji . . . (T„ is a permutation of {1 ... n}. 

2. Each node (internal or loaf) is decorated by a permutation of an interval. 

3. Each internal node labeled with -|- (resp. — ) is decorated by a permutation 
oi{i. . . j} such that its left child is decorated by a permutation oi{i . . .h— 
1} (resp. {h . . . i}) and its right child is decorated by a permutation of 
{h ■ ■ .j} (resp. {i . . .h — 1}), for some h & {i + 1, . . . , j}. 

It is easy to prove [BBL98] that to each separable permutation one can 
associate a binary separating tree (see Figure 1). Note that this tree is not 
uniquely defined as shown in figure 1. However, one can associate a unique 
tree to each separable permutation by taking arbitrary ordered trees instead 
of binary ordered trees. These trees are the contraction of the binary ones by 
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Figure 1: Two decorated binary separating trees and the decorated contracted 
separating tree ofcr = 423165897 



contracting every edge between two nodes with the same label + or — . In these 
trees, the signs of the internal nodes are thus ranked: if the root of the tree has 
label + (resp. — ), then every node at odd depth has label + (resp. — ) and 
every node at even depth label — (resp. +), so that all labels are determined 
from the label of the root. 

Definition 1.5. The unique contracted separating tree associated to a separable 
permutation a is obtained from any binary separating tree of a by contracting 
every edge between nodes of the same sign. 

1.2 Modular decomposition of graphs ; Common interval 
decomposition of permutations 

The contracted separating trees we introduced in Definition 1.5 also appear 
in graph theory. Namely, those trees are a special case of common interval 
decomposition trees (on which we however need to add a labeling). The common 
interval decomposition trees are an equivalent for permutations of the modular 
decomposition trees for graphs [dM03, BXHP]. 

Before we come to the pattern matching problem in permutations, we need to 
introduce the common interval decomposition trees, the labeled decomposition 
trees, and finally the expanded decomposition trees that are the key structure 
we use in our algorithms. 
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The common interval decomposition of a permutation a of size n (common 
meaning common with Idn) is defined as follows. First consider all the common 
intervals of a that is to say all the subsequences ajaj+i . . .ak of consecutive 
entries of cr such that {aj, cTj+i, . . . , Cfe} is an interval of N. Among the common 
intervals, the strong common intervals (or strong intervals for short) are those 
that do not overlap any other common interval^. Figure 2 illustrates the notion 
of common intervals and strong intervals. 



51 10 9678 11 243 

Figure 2: Common interval decomposition of a = 5 1 10 9 6 7 8 11 2 4 3. 
Common intervals are represented by horizontal lines, with strong intervals cor- 
responding to bold lines. 

The inclusion ordering yields a tree-like ordering on the set of strong inter- 
vals. This ordering is represented by a tree whose leaves are cti, fT2, . . ., cr„ from 
left to right in this order, whose root is cr, and such that each internal node is 
the union of its children. 

Note that there are two different types of internal nodes in the tree of strong 
intervals. For some nodes, say V, with k children V\,. . . ,Vk from left to right, 
there do not exist {i,i) ^ (l,fc) such that 1 < ?' < j < fc and the imion of 
Vi, Vi+i, ■ ■ ■ ,Vj is an interval. These nodes are called prime nodes and are of 
type P. 

The other nodes V are such that every union of consecutive children form 
an interval. Those node are called linear nodes and are of type L. 

The tree along with the types P and L of the internal nodes (see Figure 3) 
is called the common interval decomposition tree of cr. In this tree, the order of 
the children of a node depends on a so that we have an ordered tree, unlike the 
modular decomposition trees for graphs. 

Note that nodes of arity 2 verify both linear and prime definitions. We 
choose to consider them of type L. This choice will be explained later. For 
further explanation on these trees, like the proof that a node is either linear or 
prime, refer to [BXHP]. In [BCdMR, BXHP], the authors show how to compute 
these trees in linear time. 

For our algorithmic use of common interval decomposition tree, the types of 
the internal nodes are not sufficient and we need to label the internal nodes of 
the common interval decomposition tree of a permutation. 

^The definition of overlapping common intervals follows the intuition: we say that two 
common intervals I and J are overlapping when I \ J j^li, J \ I ^ 9 and I (1 J ^ 9 
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Figure 3: Common interval decomposition tree for (7 = 51 10 967811243 

It is easy to see that for each node V of type L, the intervals of values 
corresponding to the children of V are ordered citlic;! by increasing order or by 
decreasing order, when considering these children from left to right. The linear 
nodes are subsequently labeled + or — respectively. 

The labeling of a prime node with d children consists of a permutation a of 
size d that does not have any common intervals except {1},. . . ,{d} and a. Such 
permutations are known as simple permutations [AA05, AAK03, Bri, BHV06]. 
a represents the ordering of the children Vi, . . . ,Vd oi V between them with 
respect to the values in the common intervals corresponding to the V^'s. Namely, 
CTi < cTj if and only if the common interval corresponding to Vi contains values 
that are smaller than those contained in the common interval corresponding to 
Vj . For example, the simple permutation labeling the root of the tree on figure 
3 is 3 1 4 2. 

This common interval decomposition tree along with labels +,— and simple 
permutations can be computed easily in O(n^lnn) which is sufficient for our 
purpose. Remind that the common interval decomposition tree can be computed 
in linear time. Thus it remains to add a label on each internal node V. This 
can be done by sorting the intervals corresponding to the children of V. 

When a common interval decomposition tree is labeled, the common intervals 
corresponding to the nodes can be deduced from this labeling, like in the case of 
binary separating trees (see definition 1.4). So that the common intervals can 
be seen as a decoration on the nodes of the common interval decomposition tree 
with labels on internals nodes. 

Definition 1.6. The labeled decomposition tree of a permutation a is the 
common interval decomposition trcc> of a, where we add the labeling on internal 
nodes described above, and whore wc forget the decoration. 

Figure 4 gives the labeled decomposition tree of a = 5 1 10 9 6 7 8 11 2 4 3. 

We have the following nice characterization of separable permutations in 
terms of labeled decomposition trees: 
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Proposition 1.1. The separable perm,uf,ations are exactly those having a com- 
mon interval decomposition tree with no prime nodes. 

For any separable permutation, its contracted separating tree and its labeled de- 
composition tree are equal. 

The proof of this proposition is straightforward from the definition of sep- 
arable permutations. Note that it is important to give the type L to binary 
internal nodes for stating that proposition. 

Proposition 1.1 states that labeled decomposition trees are a generalization 
to all permutations of the contracted separating tree defined only for separable 
permutations. For binary separating trees, we have an analogous generalization, 
called expanded decomposition trees. In our algorithms, it is easier to work 
on binary nodes so that the natural representation used later is the expanded 
decomposition tree. 

To transform a labeled decomposition tree into an expanded decomposition 
tree, take each linear node V with children Vi,...,Vk and note that we can 
represent it by (. . . ((Vi, V2), V3), . . .), Vk) as shown in figure 4. Each positive 
(resp. negative) internal node of arity k gives k — 1 positive (resp. negative) 
binary internal nodes. 

A consequence of proposition 1.1 is: 

Proposition 1.2. The expanded decomposition tree of a separable permutation 
is one of its binary separating trees. 

2 Longest common pattern between two permu- 
tations, including one separable 

In this section, we describe a polynomial time algorithm for finding a longest 
common pattern between two permutations a and r provided that a is sepa- 
rable. This algorithm uses the same technique as the one of Bose, Buss and 
Lubiw [BBL98] for finding an occurrence of a separable pattern in a general 
permutation. Namely, it computes a binary separating tree of a and uses 
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it as a guide in the search of the longest common pattern with t. To avoid a 
complexity blow-up, the key point is to use dynamic programming, the initial 
problem being decomposed into sub-problems according to the structure of T^. 

First recall that it can be decided in linear time whether a permutation is 
separable or not. If it is, its binary separating tree can also be computed in 
linear time, as described in [BBL98]. This result is actually a special case of a 
more general one, proved in [BCdMR, BXHP], and stating that the common 
interval decomposition tree of any permutation can be computed in linear time. 

Instead of two permutations a and r, our algorithm takes as an input a 
binary separating tree T^- for a separable permutation a of size fc, and a permu- 
tation r in the usual representation rir2 . . . t„. Notice that a binary separating 
tree for a has 0{k) nodes. 

More precisely, the algorithm fills in the array 

M = {M(K a, 6) : V a node of T^, 1 < i < j < n, 1 < a < 6 < n}. 

For any node in T^, let us denote by (t(V") the subpermutation of a corre- 
sponding to the subtree of T^- rooted at V. With the notations of section 1, 
<t{V) is the permutation decorating the node V. The cell M{V,i,j, a, b) of the 
array M contains a longest common pattern tt between a{V) on one hand, and 
the subpermutation . . . tj of r on the other hand, with the additional restric- 
tion that the occurrence of tt in Tj . . . tj uses only entries of r whose values are 
between a and b. The empty pattern, of size 0, will be denoted e. 

Example 2.1. If the node V represents the pattern 2 1 (i.e. (j{V) = {i + 
1) i), and t = 642531 then we have for instance M(y, 2,4,3, 5) = 1, 
M{V, 2, 5, 3, 4) = 2 1 and M(V, 4, 5, 1, 2) = e. 

The algorithm works as follows. To start the computation, we fill in the sub- 
arrays M{V, _, _, _, _) for all the leaves V of To-. Then, we compute M{V, _, _, _, _) 
for any internal node V using the subarrays M{Vl,-, -, -, -) and M{Vr, _, _, _, _) 
corresponding to the left child (Vl) and the right child (Vr) of V. In order 
to combine the patterns found in M (Vl -) and M(Vr, _) with the 
intention of filling M(V, _, _, _, _), we need a definition of pattern concatenation. 

Definition 2.2. Consider two patterns tt and tt' of respective lengths k and k' . 
The positive and negative concatenations of tt and tt' are defined respectively by: 
TreTr' = TTi • ■■TTkiTr'i+k) ■ ■ ■ (ttj., +fc) and ttQit' = {-jri+k') ■ ■ ■ {■Kk + k')'ir[ • • - tt^/ 

Example 2.3. 

43521©3142=43521:8697 
4352163142=87965:3142 

It is clear from definition 2.2 that the positive (resp. negative) concatenation 
of two patterns produces again a pattern. 
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Algorithm 1 Longest common pattern between two permutations, first one 
separable 

1: Input: A binary separating tree T„ of a separable permutation <j of size k 
and a permutation r of size n 

2: Create an array M: 

3: for any node V of and any integers i, j, a and b between 1 and n do 

4: M{V,i,j,a,b)^e 

5: end for 

6: Fill in M for the leaves of T„: 

7: for any leaf V of T^r do 

8: for any integers «, j, a and 6 between 1 and n, i < j, a <h do 

9; if there exists some h & + 1, . . . , j} such that a < Th < b then 

10: M{V,i,j,a,b) ^1 

11: end if 

12; end for 

13: end for 

14; Fill in the rest of M: 

15: for any internal node V oiTcr, considering the nodes in the postfix ordering 
do 

16; if y is a positive node then 

17: for any integers i, j, a and b between 1 and n, i < j, a < b do 

18: M{V,i,j,a,b) ^ Longest{{M{VL,i,h-l,a,c-l)(BM{VB.,h,j,c,b) : 

i<h<j + l,a<c<b+l}) 

19: end for 

20: else 

21: /* V is a, negative node /* 

22: for any integers i, j, a and b between 1 and n, i < j , a < b do 

23: M{V,i,j,a,b) ^ L(mgest{{M{VL,i,h-l,c,b)QM{VR,hJ,a,c-l) : 

i<h<j + l,a<c<b+l}) 

24: end for 

25: end if 

26; end for 

27: Output: M(root of T^, 1, n, 1, n) 



The detailed dynamic programming algorithm is given in Algorithm 1. 

In the previous, for any set S of patterns, Longest{S) returns a longest pat- 
tern in the set S. If we want Longest to be a uniquely defined application, we 
can choose Longest{S) as the smallest pattern with respect to the lexicograph- 
ical ordering among the patterns in S having maximal length. 

Proposition 2.1. Algorithm 1 is correct: it outputs a longest common pattern 
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between the two permutations a and t given in input. 
Proof. The proof is by induction. 

We show that the algorithm stores in M{V,i,j,a,b) a longest common pattern 
between a{V) and n . . . Tj whose occurrence in r, . . . Tj uses only values between 
a and b. 

If y is a leaf, the above statement is clearly true. 

If V is an internal node, with two children Vl (its left child) and Vr (its 
right child), then let i, j, a and b be integers such that 1 < i < j < n and 
1 < a < b < n. We assume in the rest of the proof that y is a positive node, the 
case of a negative node being very similar. To begin with, it is easy to see that 
M{V, i, j, a, b) contains a common pattern between criV) and . . . Tj using only 
values between a and b in Ti.. .Tj. Indeed, by induction hypothesis, we infer 
that every pattern in the set S — {M{Vl, i,h — l,a,c — 1) (B M{Vr, h,j, c, b) : 

1 < h < j + I, a < c < b + 1}, and a fortiori Longest(S), is a common pattern 
between (t(V") and Ti . . . Tj using only values between a and b in Ti .. .Tj. 

Example 2.4. Consider the permutations (7 = 14236578 and 
T = 41325689 7. A separating tree of 

a is represented on the right. For example, we 

can choose V to be the right child of the root 

of T^. Then we have a{V) = 423657 8, 

cr(VL) = 4 2 3 and a{VR) = 6 5 7 8. Now choose 

i = 2, j = 7, a = 2 and 6=8. We want to show 

that for any h G {i, . . .j + 1} and c G {a, ... 6 + 

1}, M{VL,i,h-l,a,c- 1) ® M{VR,h,j,c,b) is a 

common pattern between cr{V) and Tj . . . tj using 

only values between a and b in n .. .Tj. Take for 14236578 

example h = 5 and c = 4. 
By induction hypothesis, M{VL,i,h — l,a,c — 1) contains a longest common 
pattern between <t{Vl) and n . . .th-i using only values between a and c — 1 
in Ti...Th-i. Here, M{VL,i,h— l,a,c— 1) = 2 1, an occurrence of 2 1 in 
Ti . . . Th-i = 13 2 using values between 2 and 3 being 13 2, and an occurrence 
of 2 1 in a-(Vt) = 4 2 3 being 4 2 3. Similarly, we have M(Vr, c, 6) = 
M(Vr, 5, 7, 4, 8) = 1 2 3, as shown by the occurrences 5 6 8 in r^j . . . Tj =568 
and 6 5 7 8 in ct(Vh) = 6 5 7 8. The occurrence of M(Vl, i, /i - 1, a, c - 1) © 
Miy^i, h,j,c,b) = 2 1 3 4 5 in Tj . . . Tj using values between a and b is thus 
obtained by considering simultaneously the two occurrences in t enlightened 
before. Namely, an occurrence of 2 1 3 4 5 in Tj ... t^ = 1 3 2 5 6 8 using values 
between 2 and 8 is 132568. Notice also that the occurrence 4236578 of 

2 1 3 4 5 in o-{V) = 4236578 is again obtained considering simultaneously 
the occurrences of 2 1 and 1 2 3 in o'{Vl) and c7(V/{) respectively. 

It remains to show that: 

Lemma 2.5. Longest{S) is of maximal length among all the common patterns 
between aiy) and Tj . . . Tj using only values between a and b in Ti. . .Tj. 
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Proof. Wc denote by tt a longest common pattern between a{V) and Tj . . .Tj, 
using only values between a and 6 in . . . . 

As shown on Figure 5, there exist integers h G {i,. . . ,i+l} and c e {a, . . . , b+1} 
such that TT is decomposed into tt = tti © 7r2 , with tti a common pattern between 
(7(Vl) and Tj . . . t?i_i, using only values between a and c — 1 in Tj . . . t/(_i, and 
772 a common pattern between a{Vii) and Th ■ ■ - tj, using only values between c 
and 6 in r/i . . . Tj. 




Notice that in this decomposition tti or TT2 might be the empty pattern. 
It can be easily seen that if tti (respectively 772) were not a pattern of maxi- 
mal length for the given intervals of indices and values, then tt would not be 
of maximal length either, contradicting the definition of tt. Consequently, tti 
and 772 are longest common patterns between <t(Vl) and (t(Vr) respectively, 
and T in the given intervals of indices and values. By induction hypothesis, 
\M{Vl, i,h — l,a,c—l)\ = \tti \ and \M{Vr, h,j, c,b)\ = |7r2|. The pattern stored 
in M{V, a, b) by the algorithm is of length at least \M{Vl, i,h—l,a,c—l)® 
M{Vr, h,j, c, 6)1 = |7ri © 7r2| = |7r|. tt being of maximal length by assumption, 
we conclude that M{V, i, j, a, b) is also of maximal length. □ 

Finally, M{V, a, b) contains a longest common pattern between aiV) and 
Tj . . . Tj whose occurrence in Tj . . . Tj uses only values between a and b. 

When is a negative node, wc decompose tt into tti B7r2, with tti a common 
pattern between oiVL) and Tj . . . Th-i , using only values between c and b in 
Ti . . .Th-i, and 772 a common pattern between a{Vii) and Th - ■ - Tj, using only 
values between a and c— 1 in Th ■■ - Tj, and the proof follows the same steps. □ 

Proposition 2.2. Algorithm 1 has a time complexity in 0{min{k,n)kn^) . 
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Proof. Algorithm 1 handles an array M of size 0{kn'^), where each cell contains 
a pattern of length at most min{n,k), so that the total space complexity is 
0{min{n, fc)fcn^). For filling in the subarrays M{V, _, _, _, _) for all the leaves 
V of Tcr (line 6 to 13 of algorithm 1), the total time complexity is 0{kn^). 
And for any internal node V, filling in one entry of the subarray M(V, _, _, _, _) 
costs 0{min{k,n)n'^), since at line 18 (or 23) of algorithm 1, we search for an 
element of maximal length among 0(n.'^) elements, each of size 0{rain{k,n)). 
Consequently, filling in completely this subarray M(y, _) requires a time 

complexity 0{min{k,n)n^). Since there are 0{k) internal nodes in T„, we have 
the announced result. □ 

This complexity can be improved to 0{knP), storing an integer, a label 
(© or 9), and two pointers in M{V,i,j,a,b) (when V is an internal node) 
instead of a pattern. Namely, if algorithm 1 fills in M{V, i,j, a, b) with the pat- 
tern p = M{VL,i, /i — 1, a, c — 1) ffi M{Vr,. h, j, c, b), it is sufficient to store in 
M{V, a, b) the length of p, the label ©, and two pointers pointing to the en- 
tries M{Vl, i,h — l,a,c—l) and M{Vr, h, j, c, b) of the array M. At the end of 
the algorithm, this system of pointers gives a binary separating tree of a longest 
common pattern tt between a and t. From this tree, tt can be computed in 
linear time [BBL98]. 

A consequence of properties 2.1 and 2.2 is: 

Theorem 2.6. The problem of finding a longest common pattern between two 
permutations, the first one being separable, is in P. 

3 Longest common pattern between two permu- 
tations 

The result of theorem 2.6 can be easily extended to classes of permutations that 
are less restricted than separable permutations. Using the common interval de- 
composition tree introduced in section 1.2, we will see that a longest common 
pattern between two permutations a and r can be computed in polynomial time 
as soon as the arity of any prime node in the common interval decomposition 
tree of a is bounded by a constant d chosen independently. 

Proposition 1.2 states that expanded decomposition trees are a generaliza- 
tion to all permutations of binary separating trees, defined only for the separable 
ones. From this remark, it becomes natural to try and use expanded decom- 
position trees in an algorithm for finding a longest common pattern between 
two general permutations. In the following, we describe such an algorithm and 
analyze its complexity: it is not a polynomial time algorithm, but the com- 
plexity analysis reveals classes of permutations for which the algorithm runs in 
polynomial time. 
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First, wc notice that the expanded decomposition tree of a permutation can 
be computed in polynomial time, and even in linear time if we have an inde- 
pendent bound on the arity of the prime nodes. [BCdMR, BXHP] provide a 
0(n)-timc algorithm for computing the common interval decomposition tree 
T„ of any permutation a of size n. This tree can easily be labeled in time 
0{d\ogd-n) where d is a bound on the arity of prime nodes: for every internal 
node V of T„ that arc in number 0{n) it is enough to sort its children to find 
the label of V. Finally, the vertical expansion necessary to obtain the expanded 
decomposition tree of a requires again a linear time. 

Algorithm 2 takes as an input an expanded decomposition tree T„ of a per- 
mutation (J and a permutation r. It outputs a longest common pattern between 
a and r. It works just like algorithm 1, except for the case of prime nodes in 
To-. The procedure in this additional case is described in algorithm 2. 



Algorithm 2 Longest common pattern between two permutations 
1: Input : An expanded decomposition tree T„ of a permutation <j of size k 
and a permutation r of size n 

2: Create an array M: proceed as in algorithm 1 

3: Fill in M for the leaves of T„: proceed as in algorithm 1 
4: Fill in the rest of M: 

5; for any internal node V oiT^, considering the nodes in the postfix ordering 
do 

6: if is a positive or a negative node then 
7; proceed as in algorithm 1 
8: else 

9: /* V is a prime node /* 
10: Let p be the simple permutation labeling V 

11: Let d be the arity of V, and Vi,. . . ,Vd the children of V, from left to 
right 

12: for any integers i, j, a and b between 1 and n, i < j , a < b do 
13: M{V, a, b) <— Longest{S) where 

S= [Qp i{M{Vk,hk-i,hk - l,Cp,._i,Cp,, - l))i<fc<^) : 

i = ho < hi < . . . < hd = j + 1, a = Co < d < . . . < hd = b + l} 
14: end for 
15: end if 
16: end for 

17: Output : M(root of T^, l,n, l,n) 



Algorithm 2 uses a more general kind of pattern concatenation than just © 
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and 0. The ^-concatenation, or pattern concatenation according to p, is defined 

as follows: 

Definition 3.1. Given p = pi . . .p„ a permutation of size n, and n patterns 
TT^, . . . , tt" of respective size fci, . . . , fc„, the p-concatenation of the (7r')i<i<„ is 

0p(7r"^, . . . , tt") = shift(7r\ pi) . . . shift(7r", p„) where 

shift(7r\pj) = shift(7r', p,;)(l) . . . shift(7r\ pi)(fci) and 
shift(7r*, Pi){x) = (7r*(x) + k^-i + . . . + k^-i ) for all x between 1 and fcj 

Example 3.2. 

©25314(21, 312, 4321, 12, 231) = 4 3 : 14 12 13 : 8 7 6 5 : 1 2 : 10 11 9. 
We can also notice that 0i2 = ® and 02i = 9- 

The idea behind algorithm 2 is quite simple. When filling in M{V,i,j,a,b) 
for a prime node V labeled by p and having d children Vi . . .Vd, we "slice" the 
intervals {i,. . . ,j} and {a,...,b} into I\. . .Id and Ai . . . Aj, respectively, such 
that Ip -< Ik and Ap -< A\? whenever p < k. Then we p-concatenate longest 
common patterns between the cr(Vfe) and t in the intervals Ik of indices and 
Ap^ of values. With the notation of algorithm 2, 1^ = {hk-i, . . . , /ife — 1} and 

M = {Cfc-l, . . . ,Cfe - 1}. 

Proposition 3.1. Algorithm 2 is correct: it outputs a longest common pattern 
between the two permutations a and r given in input. 

Proof. Similar to the proof of proposition 2.1. 

With the notations of the proof of proposition 2.1, in the case of a prime node 
V labeled by p, with children Vi . . . Vd, there exist integers i ~ < hi < . . . < 
hd = j + I and a = cq < ci < . . . < Cd = b + 1, such that we can decompose 
TT into TT = 0p(7r^, . . . , tt''), with tt*^ a common pattern between cr(Vfe) and 
Thk-i ■■ - Thk-i ) using only values between Cp^-i and Cp^ - 1 in Th,,_-, ■ . . Th^-i- 
Using this decomposition of tt, we can use the induction hypothesis on the nodes 
(^fe)i<fc<d and finish the proof as before. □ 

In this proof, the trick relies on the fact that a common pattern between 
a{V) and Ti . . . Tj is always a concatenation of common patterns between the 
children of V and "slices" of . . . tj. This stability when going from parents to 
children in the expanded decomposition tree also appears in a paper of Albert 
and Atkinson [AA05], for example in their lemma 15. 

The main difference between algorithms 1 and 2 lies in the complexity anal- 
ysis. Those two algorithms deal with dynamic programming arrays of the same 
size, but the cost for computing one entry can be very superior in algorithm 2 
than in algorithm 1. Indeed, for any internal node V, in order to fill in one entry 

^By A ^ B, we mean that Va e A, V6 6 S, a < b. 
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of M(V, _), algorithm 1 computes a longest pattern in a set containing 

0{'n?) elements, whereas in algorithm 2, the set from which we have to extract 
a longest pattern contains 0{'n?'^~'^) elements, if d is the arity of V (see line 13 
of algorithm 2) . 

With no hypothesis on a permutation a of size k, the only bound we can 
give on the maximal arity d of a prime node in the expanded decomposition 
tree of ct is d < fc. This boimd is optimal since the equality d = fc is achieved 
when (7 is a simple permutation. The total time complexity of algorithm 2 is 
consequently 0{min{n,k)k'n?''^'^), and it is not polynomial. However, if we 
consider classes of pcrmiitations such that the arity of any prime node in their 
expanded decomposition tree is bounded by a constant d, algorithm 2 has a 
time complexity 0{min{n, fc)fcn^'^+^). In this particular case, algorithm 2 runs 
in polynomial time. 

This can be summarized in the following theorem: 

Theorem 3.3. Let d be a integer. Consider the class R of permutations having 
an expanded decomposition tree with all prime nodes of arity smaller than d. 

Then the problem of finding a, longest common paitem between a permutation 
in R and another unrestricted permutation is in P. 

4 Conclusion and open problems 

We generalize the algorithm given in [BBL98] for the problem longest common 
pattern. Yet our algorithm seems far from optimal. For example, for separable 
permutations, our work is based on [BBL98] but Ibarra [Iba97] give a faster 
{0{n^)) algorithm for the pattern involvement problem. Could this algorithm 
be adapted to the longest common pattern problem? Yet a lower bound is 
given by the edit distance problem [ZS89] as the edit distance problem between 
two trees is a special case of the longest common pattern problem as shown in 
[MR06]. 
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